Smart touchscreen key activation detection

ABSTRACT

Embodiments relate to systems for, and methods of, detecting attempted space key activations on a touchscreen. Such systems and methods allow for error-tolerant data input on a touchscreen. The systems and methods may be adaptive and grow progressively more accurate as additional user data is received.

PRIORITY

This is a continuation of U.S. application Ser. No. 13/360,242, filed onJan. 27, 2012, entitled “Smart Touchscreen Key Activation Detection,”assigned to the corporate assignee of the present invention andincorporated herein by reference.

SUMMARY

According to various embodiments, a computer implemented method ofdetecting an activation of a virtual key is disclosed. The method mayinclude determining, by a computing device, a location of a thumbcontact detected at a touchscreen of the computing device, determining,by the computing device, a size of an area of the thumb contact, anddetermining an orientation of the thumb contact. The method may furtherinclude comparing the size of the area, the orientation, and thelocation to data associated with previous thumb and finger contactdetected at the touchscreen. The method may further include determining,based on the comparing, that the thumb contact constitutes an intendedactivation of a given virtual key, and outputting, in response to thedetermining, an electronic representation of a character associated withthe given virtual key.

DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the presentteachings and together with the description, serve to explain theprinciples of the present teachings. In the figures:

FIG. 1 is a schematic diagram of an apparatus according to variousembodiments;

FIG. 2 is a schematic diagram of a touchscreen keyboard layout accordingto various embodiments;

FIG. 3 is a schematic representation of the contacts of a human user'sdigits when touch typing according to various embodiments; and

FIG. 4 is a flowchart of a method according to various embodiments.

DESCRIPTION OF EMBODIMENTS

Devices that employ touchscreens may use the space key as, for example,an enter or confirmation button to activate predicative word completion.Accordingly, such touchscreen devices tend to depend on absoluteaccuracy in space key activations (even if such demands are not made ofthe other keys). However, a need for absolute accuracy driven bypredicative word completion makes it difficult to correctly interpretspace key activation attempts that miss the area of the touchscreendesignated as the space key. Accordingly, there is a need for smartspace key activation detection for touchscreen devices.

Embodiments are not limited to space key activation attempt detection.In general, devices that employ touchscreens are amenable to usingdisclosed embodiments to detect activation attempts of any particularkey, where one or both of the following obtain: the particular key istypically activated by a particular digit or portion of the hand, andthe particular key is located in a particular region of the touchscreen.An exemplary such key, as discussed above, is the space key. As anotherexample, users may activate the enter key using a knuckle or otherportion of a hand, and the enter key is located at a particular regionof the touchscreen. Accordingly, the enter key is amenable to thetechniques disclosed herein. Embodiments disclosed herein are notlimited to space keys and enter keys.

Various embodiments directed to space key activation detection track athumb signal for each possible space key activation. The thumb signal isintended to differentiate between thumb touches and touches by fingers,as thumb touches tend to correspond to space key activations. (Note thatthroughout this disclosure, the word “finger” refers only to non-thumbhand digits.) The thumb signal may include any, or a combination, ofthree components: a touch area size, a touch area orientation, and atouch area texture. Touch area size measures the size of the area ofcontact between the touchscreen and digit, which tends to be larger forthumbs than for fingers. Touch area orientation accounts for the factthat thumb touches tend to be oblong in the shape of the contact area,whereas touches by other digits tend to be more circular. Touch areatexture measures qualities such as ridges and joins splits, which aredistinct between thumbs and fingers.

Various embodiments directed to detecting space key activation attemptsmay in addition, or in the alternative, track a location signal for eachpossible space key activation. The location signal is intended toexploit the fact that statistically significant lower contacts (that is,low on the part of the touchscreen designated as a keyboard) near intime to higher contacts tend to be attempted space key activations.

Various embodiments directed to detecting space key activation attemptsuse either or both of the thumb signal and the location signal todetermine whether a possible space key activation should be consideredan actual space key activation attempt. Such embodiments may useadaptive learning techniques to compare one or both of the thumb signaland the location signal of a possible space key activation to suchsignals from past actions. Thus, such embodiments use adaptive learningto differentiate between space key and other key activation attempts.

Reference will now be made in detail to exemplary embodiments of thepresent teachings, which are illustrated in the accompanying drawings.Where possible the same reference numbers will be used throughout thedrawings to refer to the same or like parts.

FIG. 1 is a schematic diagram of an apparatus according to variousembodiments. Apparatus 102 may be any computing device, such as, by wayof non-limiting example, a smart phone, a personal digital assistant(PDA) or a tablet computer. Apparatus 102 may be portable or a fixture.In FIG. 1, apparatus 102 includes or is communicatively coupled totouchscreen 104. Touchscreen 104 may be positioned in a manner such thata user of device 102 can physically interact with touchscreen 104.

Touchscreen 104 may implement any of a variety of technologies. By wayof non-limiting example, touchscreen 104 may be any of the followingtypes: resistive, surface acoustic wave, capacitive (e.g., surfacecapacitance, projected capacitance, mutual capacitance orself-capacitance), infrared, optical imaging, dispersive signaltechnology or acoustic pulse recognition. In FIG. 1, touchscreen 104 iscommunicatively coupled to processor 106. Processor 106 may be, by wayof non-limiting example, a microprocessor or a microcontroller. In FIG.1, Processor 106 is capable of carrying out electronically storedprogram instructions and is communicatively coupled to persistent memory108. Persistent memory 108 may include, by way of non-limiting example,a hard drive or a flash memory device.

FIG. 2 is a schematic diagram of a touchscreen keyboard layout 202according to various embodiments. In particular, FIG. 2 depicts a QWERTYkeyboard layout, although embodiments are not so limited (e.g.,embodiments may include a DVORAK keyboard layout, a numeric keypadlayout, or another type of keyboard or keypad layout instead). Asdepicted in FIG. 2, keyboard layout 202 includes a section for upperkeys 204 and space key 206. (Throughout this disclosure, “space key” issynonymous with “spacebar”.) As depicted in FIG. 2, space key 206 ispositioned apart from upper keys 204 such that space key 206 isgeometrically lower than upper keys 204.

Touchscreen keyboard layout 202 may be implemented in, by way ofnon-limiting example, touchscreen 104 of FIG. 1. That is, touchscreenkeyboard layout 202 may be displayed on touchscreen 104 of FIG. 1, andthe areas demarcated as keys may have the corresponding key functionassociated with them by processor 106 of the device of FIG. 1.

FIG. 3 is a schematic representation of the contacts of a human user'sdigits when touch typing according to various embodiments. Inparticular, when touch typing, a human user is likely to positionleft-hand fingertip contacts 302 relatively higher than left-hand thumbcontact 306. Likewise, right-hand fingertip contacts 304 are typicallypositioned relatively higher than right-hand thumb contact 308. Eachfingertip contact 302, 304 typically occupies less area than thumbcontacts 306, 308.

Each finger and thumb contact 302, 304, 306, 308 has an associateddiameter, which may be defined as the length of the longest line segmentthat can be contained in the periphery of the contact. FIG. 3illustrates the diameter of thumb contact 306 as the longest linesegment 310 that can be fit into the periphery of thumb contact 306.Typically, the diameters of thumb contacts 306, 308 are significantlygreater than the diameters of finger contacts 302, 304. (Fingertipcontacts 302, 304 are typically substantially more circular than thumbcontacts 306, 308, which themselves are substantially more oval.)

Each finger and thumb contact 302, 304, 306, 308 has an associatedorientation, which may be defined as the angle between anupward-pointing ray and the diameter line segment. Note that theorientation may be defined as ranging from 0° to 180° by restricting theangle measurement to the first and fourth quadrants of the Cartesianplane, or may equivalently be defined as ranging from −90° to +90° byrestricting the angle measurement to the first and second quadrants ofthe Cartesian plane. FIG. 3 illustrates the orientation θ of thumbcontact 306. More particularly, FIG. 3 depicts the orientation of thumbcontact 306 as the angle θ between diameter 310 and upward-pointing ray312 as measured in the first quadrant of the Cartesian plane. Typically,the orientation of thumb contacts 306, 308 is consistently differentfrom that of finger contacts 302, 304.

Each contact 302, 304, 306, 308 has an associated texture. The textureincludes qualities such as the presence, location and orientation offriction ridges and joint creases. Texture may be determined using, byway of non-limiting example, a high-resolution optical touchscreen.Typically, the texture of thumb contacts 306, 308 is consistentlydifferent from that of finger contacts 302, 304. By way of non-limitingexample, thumb contacts 306, 308 typically include a joint crease,whereas finger contacts 302, 304 typically do not.

FIG. 4 is a flowchart of a method according to various embodiments. Themethod depicted in FIG. 4 may be implemented using, for example, thedevice discussed above in reference to FIG. 1. The method discussed inreference to FIG. 4 may be user-specific, as different users may beassociated with different historical touchscreen contact data andmathematical models based thereon. For example, a user's log-in profilemay be associated with, or capture and store, such data. Embodiments mayallow a user to opt out of having such individualized data collected andstored.

At block 400, historical touchscreen contact data are collected andstored. The historical touchscreen contact data can include, for anumber of finger and thumb contacts, data representing contact areasize, contact orientation, contact texture and contact relativeposition. The number of contacts for which data is collected at thisblock may be, by way of non-limiting example, 10, 20, 30, 40, 50, 75,100, 150, 200, 250, 500 or 1000. Specific techniques for detecting andconverting the detected contact information into stored valuesrepresenting contact area size, contact orientation, contact texture andcontact relative position are discussed below in reference to blocks402-408. The historical contact data may be stored in, e.g., persistentmemory 108 of device 102 of FIG. 1.

Each datum in the historical contact data may be associated with anidentifier as to whether the datum arose from a finger contact or athumb contact. For example, touchscreen contacts that activate upperkeys (those keys 204 of FIG. 2) may be associated with finger contacts,while touchscreen contacts that activate the space key (206 of FIG. 2)may be associated with thumb contacts. (For the calibration phase, onlyexact activation contacts may be considered; near misses may beexcluded.) Thus, each of a plurality of touchscreen contacts may beassociated with a finger contact or a thumb contact and the relevantdata stored.

The historical touchscreen contact data may be collected during normaluser interactions or during a dedicated calibration phase. The dedicatedcalibration phase, if used, may include displaying a message to a userrequesting that the user type some pre-selected text (e.g., “the quickbrown fox jumped over the lazy dog”). Alternately, or in addition, thecalibration phase may request that the user type random characters orother text.

Blocks 402-414 are directed to detecting and evaluating whether a singletouchscreen contact should be interpreted as a space key activation. Inwhat follows, such a single touchscreen contact is referred to as the“possible space key contact.”

At block 402, the contact area size of the possible space key contact isdetected. The contact area size may be automatically reported bycapacitance and other types of touchscreens. Alternately, or inaddition, the contact area size may be determined by, for example,calculating a diameter of the contact area, calculating the length of aline segment internal to the contact area that is perpendicular to thediameter, and then using the formula for the area of an ellipse based onmajor and minor axes of the ellipse. That is, if M is the diameter and mis the length of the perpendicular contained line segment, then thecontact area A may be calculated as, by way of non-limiting example,A=¼πMN.

At block 404, the contact area orientation is determined. This may beperformed by, for example, first determining the diameter, and thendetermining the relative angle between the diameter and anupward-pointing ray (e.g., 310 and 312 of FIG. 3). The diameter may bedetermined by first determining the perimeter of the contact area andthen fitting the longest contained line segment. Once the diameter isdetermined, the orientation may then be calculated.

At block 406, the contact texture is determined. The contact textureinformation may optionally include whether and where a joint creaseand/or one or more friction ridges is present in the touchscreencontact, along with any additional characteristics of the same.Biometric automatic fingerprint analytic techniques, known to those ofskill in the art, may be employed to convert the detected geometrictexture data to numerical quantities for storage.

At block 408, the relative position of the touchscreen contact isdetected. This data may include absolute positions on the touchscreenitself relative to a fixed set of x and y-axes. In some embodiments, thecentroid of the contact may be determined and recorded.

At block 410, parameters representing the detected touchscreen contactare compared to the stored historical touchscreen contact data. Severaldifferent techniques may be used to perform the comparison, as discussedimmediately below.

The comparison of block 410 may include or rely on the generation of oneor more mathematical models. In general, each contact area size, contactorientation, contact texture and contact relative position included inthe stored historical touchscreen contact data may each be considered tobe normally distributed for each of two populations: space key contactsand upper key contacts. Accordingly, statistical models (e.g., normalprobability distribution functions) may be specified for each population(space key contacts and upper key contacts) for each parameter (contactarea size, contact orientation, contact texture and contact relativeposition). Such models may be generated based on the historical datacollected at block 400. Baysean statistics in general, and maximumlikelihood estimation (MLE) in particular, may be used to form and testthe models. Thus, each parameter and each population may be associatedwith a probability distribution. MLE may produce the models by, in part,calculating a variance and mean for each population (space key contactsand upper key contacts) and for each parameter (contact area size,contact orientation, contact texture and contact relative position). Asknown to those of skill in the art, a normal probability distributionfunction may be determined by a given variance and mean.

At block 410, the parameters for the possible space key contact arecompared to the parameters predicted by the mathematical models. Ingeneral, for each parameter of the possible space key contact, aprobability may be determined as to whether it is a member of the spacekey contact population, and a related probability may be generated as towhether it is a member of the upper key contact population. Thus, one,two, three or four probabilities (depending on the number of parametersused from among contact area size, contact orientation, contact textureand contact relative position) may be associated with the probable spacekey contact. These are referred to herein as “parameter probabilities”for the respective populations.

For each population, the parameter probabilities for the possible spacekey contact may be combined into a single overall score as to whetherthe possible space key contact is in that population. The overall scorefor the possible space key contact being in the space key contactpopulation may be achieved by, for example, summing the space keycontact parameter probabilities for the possible space key contact.Similarly, the overall score for the possible space key contact being inthe upper key contact population may be achieved by, for example,summing the upper key contact parameter probabilities for the possiblespace key contact. Other techniques for combining the parameterprobabilities to achieve overall scores for space key versus upper keycontacts are possible (e.g., arithmetic or geometric mean). In someembodiments, a single overall score representing a probability ofbelonging to one of the populations (space key contacts or upper keycontacts) may be generated.

At block 412, based on the overall probability calculated at block 410,the possible space key contact may be classified as belonging to thespace key contact population or the upper key contact population. Forexample, if the overall score for the possible space key contact beingin the space key contact population exceeds the overall score for thepossible space key contact being in the upper key contact population,then the determination at block 412 may be that the possible space keycontact represents a true space key contact attempt. Likewise, if theoverall score for the possible space key contact being in the upper keycontact population exceeds the overall score for the possible space keycontact being in the space key contact population, then thedetermination at block 412 may be that the possible space key contactdoes not represent a space key contact attempt.

For embodiments that calculated a single overall score representing aprobability of belonging to one of the populations (space key contactsor upper key contacts) the determination of block 412 may include adetermination as to whether the single overall score is greater than orless than 50%. The determination of block 412 may be based on thatsingle probability.

At block 414, a space key character is output if the determination atblock 412 indicates a true space key contact attempt. This output may bemade to a keyboard buffer, or directly to an application that iscommunicatively coupled to the touchscreen at the time of the possiblespace key activation. If no true space key contact attempt is determinedat block 412, an embodiment may output nothing, or may output thecharacter represented by the nearest non-space key.

Note that a possible space key contact may be considered an actual spacekey contact attempt even if the contact misses the portion of thetouchscreen that is designated as a space key, as long as the possiblespace key contact is probabilistically more similar to the space keycontact population than it is to the upper key contact population asdiscussed above.

Note that the determination made at block 412 may be combined with theother parameters from the possible space key activation, and theresulting data added to the historical data collected at block 400. Inthis manner, embodiments may adaptively learn from repeated touchscreeninteractions. If the user subsequently deletes the character output atblock 414, embodiments may alter the resulting data to reflect thecharacter subsequently typed by the user. For example, if an embodimentoutputs a space character at block 414, and if the user subsequentlydeletes the space character and replaces it with a non-space character,then the data collected in relation to the possible space key contactmay be classified as a non-space-key contact for purposes of storagewith the historical touchscreen data discussed in reference to block400. Likewise, if a non-space key character is output at block 414 butthen deleted by the user and replaced with a space character, then theassociated data may be stored as being in the space key contactpopulation.

While the present disclosure at times discusses embodiments in which thevirtual key for which activation attempts are detected is the space key,some embodiments are not so limited. In particular, embodiments may bedirected to any virtual key that is intended to be contacted by a thumb(as opposed to fingers), or that is positioned geometrically lower on avirtual keyboard than substantially all of the remaining keys.

The foregoing description is illustrative, and variations inconfiguration and implementation may occur to persons skilled in theart. Other resources described as singular or integrated can inembodiments be plural or distributed, and resources described asmultiple or distributed can in embodiments be combined. The scope of thepresent teachings is accordingly intended to be limited only by thefollowing claims.

What is claimed is:
 1. A method comprising: determining, by a computingdevice, a location of a thumb contact detected at a touchscreen, whereinthe thumb contact detected at the touchscreen is not detected at aregion of the touchscreen associated with a virtual space key;determining, by the computing device, based at least in part on thelocation of the thumb contact detected at the touchscreen and one ormore parameters associated with the thumb contact, that the thumbcontact constitutes an intended activation of the virtual space key; andin response to determining that the thumb contact constitutes theintended activation of the virtual space key, outputting, by thecomputing device and for display, an electronic representation of aspace character.
 2. The method of claim 1, wherein the one or moreparameters associated with the thumb contact comprise one or more of asize of an area of the thumb contact, an orientation of the thumbcontact, and a texture of the thumb contact.
 3. The method of claim 1,further comprising: comparing, by the computing device, the location ofthe thumb contact and the one or more parameters associated with thethumb contact to data associated with at least one previous contactdetected at the touchscreen, wherein determining that the thumb contactconstitutes the intended activation of the virtual space key furthercomprises determining, based at least in part on the comparing, that thethumb contact constitutes the intended activation of the virtual spacekey.
 4. The method of claim 1, further comprising: assigning, by thecomputing device, a likelihood score to the location of the thumbcontact and each of the one or more parameters associated with the thumbcontact, wherein determining that the thumb contact constitutes theintended activation of the virtual space key further comprisesdetermining, based at least in part on each of the assigned likelihoodscores, that the thumb contact constitutes the intended activation ofthe virtual space key.
 5. The method of claim 1, wherein the one or moreparameters associated with the thumb contact comprise a size of an areaof the thumb contact, the method further comprising determining the areaof the thumb contact using a centroid of the thumb contact and a radiusof the thumb contact detected at the touchscreen.
 6. The method of claim1, wherein the one or more parameters associated with the thumb contactcomprise an orientation of the thumb contact, the method furthercomprising determining the orientation of the thumb contact using anorientation of a line segment comprising a maximal diameter of the thumbcontact detected at the touchscreen.
 7. The method of claim 1, whereinthe one or more parameters associated with the thumb contact comprise atexture of the thumb contact detected at the touchscreen, the methodfurther comprising determining the texture of the thumb contact usingthumbprint ridges of the thumb contact detected at the touchscreen.
 8. Asystem comprising: a touchscreen configured to display a representationof a keyboard; and at least one processor configured to: determine alocation of a thumb contact detected at the touchscreen, wherein thethumb contact detected at the touchscreen is not detected at a region ofthe touchscreen associated with a virtual space key; determine, based atleast in part on the location of the thumb contact detected at thetouchscreen and one or more parameters associated with the thumbcontact, that the thumb contact constitutes an intended activation ofthe virtual space key; and in response to determining that the thumbcontact constitutes the intended activation of the virtual space key,output, for display at the touchscreen, an electronic representation ofa space character.
 9. The system of claim 8, wherein the one or moreparameters associated with the thumb contact comprise one or more of asize of an area of the thumb contact, an orientation of the thumbcontact, and a texture of the thumb contact.
 10. The system of claim 8,wherein the at least one processor is further configured to compare thelocation of the thumb contact and the one or more parameters associatedwith the thumb contact to data associated with at least one previouscontact detected at the touchscreen, and wherein the at least oneprocessor is configured to determine that the thumb contact constitutesthe intended activation of the virtual space key by at least beingconfigured to determine, based at least in part on the comparing, thatthe thumb contact constitutes the intended activation of the virtualspace key.
 11. The system of claim 8, wherein the at least one processoris further configured to assign a likelihood score to the location ofthe thumb contact and each of the one or more parameters associated withthe thumb contact, and wherein the at least one processor is configuredto determine that the thumb contact constitutes the intended activationof the virtual space key by at least being configured to determine,based at least in part on each of the assigned likelihood scores, thatthe thumb contact constitutes the intended activation of the virtualspace key.
 12. The system of claim 8, wherein the one or more parametersassociated with the thumb contact comprise a size of an area of thethumb contact, and wherein the at least one processor is furtherconfigured to determine the area of the thumb contact by at least beingconfigured to determine the area of the thumb contact using a centroidof the thumb contact and a radius of the thumb contact detected at thetouchscreen.
 13. The system of claim 8, wherein the one or moreparameters associated with the thumb contact comprise an orientation ofthe thumb contact, and wherein the at least one processor is furtherconfigured to determine the orientation of the thumb contact by at leastbeing configured to determine the orientation of the thumb contact usingan orientation of a line segment comprising a maximal diameter of thethumb contact detected at the touchscreen.
 14. The system of claim 8,wherein the one or more parameters associated with the thumb contactcomprise a texture of the thumb contact detected at the touchscreen, andwherein the at least one processor is further configured to determinethe texture of the thumb contact by at least being configured todetermine the texture of the thumb contact using thumbprint ridges ofthe thumb contact detected at the touchscreen.
 15. A non-transitoryprocessor-readable medium storing code representing instructions that,when executed by a processor, cause the processor to: determine alocation of a thumb contact detected at a touchscreen, wherein the thumbcontact detected at the touchscreen is not detected at a region of thetouchscreen associated with a virtual space key; determine, based atleast in part on the location of the thumb contact detected at thetouchscreen and one or more parameters associated with the thumbcontact, that the thumb contact constitutes an intended activation ofthe virtual space key; and in response to determining that the thumbcontact constitutes the intended activation of the virtual space key,output, for display, an electronic representation of a space character.16. The non-transitory processor-readable medium of claim 15, whereinthe one or more parameters associated with the thumb contact compriseone or more of a size of an area of the thumb contact, an orientation ofthe thumb contact, and a texture of the thumb contact.
 17. Thenon-transitory processor-readable medium of claim 15, further comprisinginstructions that, when executed by the processor, cause the processorto: compare the location of the thumb contact and the one or moreparameters associated with the thumb contact to data associated with atleast one previous contact detected at the touchscreen, wherein theinstructions that, when executed by the processor, cause the processorto determine that the thumb contact constitutes the intended activationof the virtual space key further comprise instructions that, whenexecuted by the processor, cause the processor to determine, based atleast in part on the comparing, that the thumb contact constitutes theintended activation of the virtual space key.
 18. The non-transitoryprocessor-readable medium of claim 15, further comprising instructionsthat, when executed by the processor, cause the processor to: assign alikelihood score to the location of the thumb contact and each of theone or more parameters associated with the thumb contact, wherein theinstructions that, when executed by the processor, cause the processorto determine that the thumb contact constitutes the intended activationof the virtual space key further comprise instructions that, whenexecuted by the processor, cause the processor to determine, based atleast in part on each of the assigned likelihood scores, that the thumbcontact constitutes the intended activation of the virtual space key.19. The non-transitory processor-readable medium of claim 15, whereinthe one or more parameters associated with the thumb contact comprise asize of an area of the thumb contact, the processor-readable storagemedium further comprising instructions that, when executed by theprocessor, cause the processor to determine the area of the thumbcontact using a centroid of the thumb contact and a radius of the thumbcontact detected at the touchscreen.
 20. The non-transitoryprocessor-readable medium of claim 15, wherein the one or moreparameters associated with the thumb contact comprise an orientation ofthe thumb contact, the processor-readable storage medium furthercomprising instructions that, when executed by the processor, cause theprocessor to determine the orientation of the thumb contact using anorientation of a line segment comprising a maximal diameter of the thumbcontact detected at the touchscreen.